In [9]:
layered_view(conv2d_128,
legend=True,
spacing=20
)
Out[9]:
NAME: EDWARD TAN YUAN CHONG
CLASS: DAAA/FT/2B/04
ADM NO.: 2214407
These are the modules that are required for this notebook.
# Standard library imports
import math
import os
import time
# Third-party library imports for data manipulation and analysis
import numpy as np
import pandas as pd
# Third-party library imports for data visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Third-party library imports for deep learning
import tensorflow as tf
from keras.models import load_model
from sklearn.metrics import classification_report
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.utils import to_categorical
from visualkeras import layered_view
These are some seaborn customizations for the charts to be plotted later on.
# Change theme of charts
sns.set_theme(style='darkgrid')
# Variable for color palettes
color_palette = sns.color_palette('muted')
We will only be importing the test data to verify that our model and its weights are correctly imported later on. Following the DELE CA1 assignment for Part A, the images are resized to 128x128 and 31x31, and are grayscaled as well.
# The two different image sizes
image_size_128 = (128,128)
image_size_31 = (31,31)
def load_all_images(image_size, dataset_type):
# Normalization
datagen = ImageDataGenerator(rescale=1./255)
# Load image from directory
generator = datagen.flow_from_directory(
f"Dataset for CA1 part A/{dataset_type}",
target_size=image_size,
color_mode='grayscale',
batch_size=64,
class_mode='binary',
shuffle=False
)
# List to store all the images and labels
all_images = []
all_labels = []
# Calculate the number of loops needed to load all images and labels
num_batches = len(generator)
# Loop the number of loops needed and append each batch to the list
for i in range(num_batches):
batch_images, batch_labels = next(generator)
all_images.extend(batch_images)
all_labels.extend(batch_labels)
# Return NumPy arrays
return np.array(all_images), np.array(all_labels)
# Load datasets for 128x128 image size
X_test_128, y_test_128 = load_all_images(image_size_128, 'test')
# Load datasets for 31x31 image size
X_test_31, y_test_31 = load_all_images(image_size_31, 'test')
# Print length of each train, test, validation dataset
print(f"\n\nImage size: {image_size_128}")
print(f'Length of test array: {len(X_test_128)}.\n\n')
print(f'Length of test_label array: {len(y_test_128)}\n')
print("-"*50)
print(f"\n\nImage size: {image_size_31}")
print(f'Length of test array: {len(X_test_31)}.\n\n')
print(f'Length of test_label array: {len(y_test_31)}.')
Found 3000 images belonging to 15 classes. Found 3000 images belonging to 15 classes. Image size: (128, 128) Length of test array: 3000. Length of test_label array: 3000 -------------------------------------------------- Image size: (31, 31) Length of test array: 3000. Length of test_label array: 3000.
We will also need to one-hot encode labels for evaluation later on, which is done with tensorflow keras utility library to_categorical().
# One-hot encode y_test
y_test = to_categorical(y=y_test_128)
We dynamically obtain the labels of each class from the file in case another developer's file structure is different and may lead to a different label encoding if it were done manually.
# File directory
directory_path = '../ca2-daaa2b04-2214407-edward/Dataset for CA1 part A/test'
# Get a list of file names in the directory
file_names = [name for name in os.listdir(directory_path) if os.path.isdir(os.path.join(directory_path, name))]
# Sort and enumerate the file names to create a label encoding
labels = {idx: name for idx, name in enumerate(sorted(file_names))}
# Print result
print(labels)
{0: 'Bean', 1: 'Bitter_Gourd', 2: 'Bottle_Gourd', 3: 'Brinjal', 4: 'Broccoli', 5: 'Cabbage', 6: 'Capsicum', 7: 'Carrot', 8: 'Cauliflower', 9: 'Cucumber', 10: 'Papaya', 11: 'Potato', 12: 'Pumpkin', 13: 'Radish', 14: 'Tomato'}
Now, we will import the Convolutional Neural Network (CNN) models from the DELE CA1 Assignment, importing the weights and evaluating the models to ensure the results are up to par.
# Confusion matrix of results
def confusion_matrix(y_test, y_pred, title):
plt.figure(figsize=(14,14))
sns.heatmap(
tf.math.confusion_matrix(y_test, y_pred, num_classes=15),
annot=True,
xticklabels=labels.values(),
yticklabels=labels.values(),
cmap='icefire',
fmt='d'
).set(title='Confusion Matrix' + title, xlabel='Predicted', ylabel='Actual')
plt.show()
# Error analysis images plot
def error_analysis(X_test, y_test, y_pred, labels, title):
# Get incorrect predictions
data = {'y_test': y_test, 'y_pred': y_pred}
y_df = pd.DataFrame(data)
incorrect_predictions = y_df[y_df['y_test'] != y_df['y_pred']][:30]
# Number of rows
n_rows = math.ceil(len(incorrect_predictions) / 5)
fig, axes = plt.subplots(n_rows, 5, figsize=(20, 20))
axes = axes.flatten()
# Plotting the incorrect predictions
for idx, (index, row) in enumerate(incorrect_predictions.iterrows()):
ax = axes[idx]
ax.imshow(X_test[index], cmap='gray')
ax.set_title(f"Actual: {labels[row['y_test']]}\nPredicted: {labels[row['y_pred']]}")
ax.axis('off')
# Turn off axes for any unused subplots
for ax in axes[len(incorrect_predictions):]:
ax.axis('off')
# Title
fig.suptitle(t="Error analysis of"+title, y=0.91)
plt.show()
load_model() function to load my DELE CA1 CNN best model for 128x128 images, which was a custom built convolutional neural network. conv2d_128 = load_model("./model_weights/conv2d_128.h5")
.summary() on the model to ensure that it has been successfully loaded and view its architecture.conv2d_128.summary()
Model: "FinalConv2D128"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 128, 128, 1)] 0
conv2d (Conv2D) (None, 125, 125, 64) 1088
max_pooling2d (MaxPooling2 (None, 62, 62, 64) 0
D)
dropout (Dropout) (None, 62, 62, 64) 0
conv2d_1 (Conv2D) (None, 60, 60, 128) 73856
max_pooling2d_1 (MaxPoolin (None, 30, 30, 128) 0
g2D)
dropout_1 (Dropout) (None, 30, 30, 128) 0
conv2d_2 (Conv2D) (None, 28, 28, 256) 295168
max_pooling2d_2 (MaxPoolin (None, 14, 14, 256) 0
g2D)
dropout_2 (Dropout) (None, 14, 14, 256) 0
conv2d_3 (Conv2D) (None, 12, 12, 512) 1180160
max_pooling2d_3 (MaxPoolin (None, 6, 6, 512) 0
g2D)
dropout_3 (Dropout) (None, 6, 6, 512) 0
global_average_pooling2d ( (None, 512) 0
GlobalAveragePooling2D)
dense (Dense) (None, 128) 65664
dropout_4 (Dropout) (None, 128) 0
dense_1 (Dense) (None, 15) 1935
=================================================================
Total params: 1617871 (6.17 MB)
Trainable params: 1617871 (6.17 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
layered_view(conv2d_128,
legend=True,
spacing=20
)
Used .predict() to obtain predictions of the final model on the testing data. This allows us to use its predictions to verify the model and its weights have been successfully imported correctly.
y_pred_128 = conv2d_128.predict(X_test_128)
y_pred_128
94/94 [==============================] - 15s 157ms/step
array([[9.9998784e-01, 4.8009301e-06, 1.9927766e-11, ..., 4.2416320e-10,
2.6031596e-10, 1.5276635e-08],
[1.0000000e+00, 2.7679690e-12, 1.3531709e-18, ..., 1.2633899e-16,
5.6432236e-19, 2.4937380e-10],
[9.7311324e-01, 7.4708294e-03, 7.6567261e-08, ..., 2.1213140e-05,
2.2532136e-07, 1.5195239e-04],
...,
[5.3552746e-10, 5.9829536e-10, 8.6132324e-18, ..., 2.1695321e-11,
1.5883395e-10, 9.9999988e-01],
[2.9603275e-04, 2.6420665e-07, 7.3460378e-12, ..., 2.1851686e-06,
6.5188260e-08, 9.9969459e-01],
[3.6148336e-03, 2.2181816e-06, 7.7026364e-08, ..., 2.5347604e-03,
1.3724868e-06, 3.9216205e-01]], dtype=float32)
Utilized Scikit-Learn's classification_report() to get a classification report of the model's performance, having metrics such as Precision, Recall, and F1-score.
Precision: Ratio of true positives to total number of positive predictions [MINIMIZE FALSE POSITIVES]
Recall: Ratio of true positives to total number of actual positive predictions [MINIMIZE FALSE NEGATIVES]
F1 score: Combines precision and recall scores
# Get binary label using np.argmax() to get index. E.g. np.argmax([0,0,0,1]) will return binary value 3
y_test_binary, y_pred_128_binary = np.argmax(y_test, axis=1), np.argmax(y_pred_128, axis=1)
report_128 = classification_report(y_test_binary, y_pred_128_binary, target_names=labels.values())
print(report_128)
precision recall f1-score support
Bean 0.98 0.99 0.98 200
Bitter_Gourd 0.99 0.98 0.99 200
Bottle_Gourd 0.99 1.00 0.99 200
Brinjal 0.99 0.99 0.99 200
Broccoli 0.97 0.98 0.98 200
Cabbage 0.98 0.99 0.99 200
Capsicum 0.98 0.98 0.98 200
Carrot 0.98 0.98 0.98 200
Cauliflower 0.96 0.98 0.97 200
Cucumber 0.98 1.00 0.99 200
Papaya 1.00 0.97 0.98 200
Potato 0.99 0.98 0.99 200
Pumpkin 0.99 0.97 0.98 200
Radish 0.99 0.97 0.98 200
Tomato 0.99 0.97 0.98 200
accuracy 0.98 3000
macro avg 0.98 0.98 0.98 3000
weighted avg 0.98 0.98 0.98 3000
Plotted a confusion matrix to visualize how well the model predicts each label of vegetables.
confusion_matrix(y_test=y_test_binary, y_pred=y_pred_128_binary, title=' of Conv2D on 128x128 Images')
For the error analysis, I plotted the first 30 incorrect predictions made by the model to visualize where the model went wrong in its vegetable prediction.
error_analysis(y_test=y_test_binary,
y_pred=y_pred_128_binary,
title=' of Conv2D on 128x128 Images',
X_test=X_test_128,
labels=labels
)
ANALYSIS:
load_model() function to load my DELE CA1 CNN best model for 31x31 images, which was a custom adjusted VGG-16 model.customvgg_31 = load_model("./model_weights/vgg_31.h5")
.summary() on the model to ensure that it has been successfully loaded and view its architecture.customvgg_31.summary()
Model: "CustomVGG31"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 31, 31, 1)] 0
conv2d (Conv2D) (None, 31, 31, 64) 320
batch_normalization (Batch (None, 31, 31, 64) 256
Normalization)
conv2d_1 (Conv2D) (None, 31, 31, 64) 16448
batch_normalization_1 (Bat (None, 31, 31, 64) 256
chNormalization)
max_pooling2d (MaxPooling2 (None, 15, 15, 64) 0
D)
conv2d_2 (Conv2D) (None, 15, 15, 128) 32896
batch_normalization_2 (Bat (None, 15, 15, 128) 512
chNormalization)
conv2d_3 (Conv2D) (None, 15, 15, 128) 65664
batch_normalization_3 (Bat (None, 15, 15, 128) 512
chNormalization)
max_pooling2d_1 (MaxPoolin (None, 7, 7, 128) 0
g2D)
conv2d_4 (Conv2D) (None, 7, 7, 256) 131328
batch_normalization_4 (Bat (None, 7, 7, 256) 1024
chNormalization)
conv2d_5 (Conv2D) (None, 7, 7, 256) 262400
batch_normalization_5 (Bat (None, 7, 7, 256) 1024
chNormalization)
max_pooling2d_2 (MaxPoolin (None, 3, 3, 256) 0
g2D)
flatten (Flatten) (None, 2304) 0
dense (Dense) (None, 256) 590080
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 128) 32896
dropout_1 (Dropout) (None, 128) 0
dense_2 (Dense) (None, 15) 1935
=================================================================
Total params: 1137551 (4.34 MB)
Trainable params: 1135759 (4.33 MB)
Non-trainable params: 1792 (7.00 KB)
_________________________________________________________________
layered_view(customvgg_31,
legend=True,
spacing=20
)
Used .predict() to obtain predictions of the final model on the testing data. This allows us to use its predictions to verify the model and its weights have been successfully imported correctly.
y_pred_31 = customvgg_31.predict(X_test_31)
y_pred_31
1/94 [..............................] - ETA: 19s94/94 [==============================] - 3s 26ms/step
array([[1.00000000e+00, 8.34048596e-13, 1.93723226e-27, ...,
2.06628182e-21, 5.19481915e-18, 3.83389914e-14],
[1.00000000e+00, 4.04287239e-19, 1.84582896e-34, ...,
1.28685198e-27, 9.09532137e-22, 6.96544702e-19],
[9.99999881e-01, 1.23296600e-08, 2.60949381e-22, ...,
2.08501374e-15, 2.88106028e-16, 3.02837443e-12],
...,
[6.87276909e-14, 8.91513890e-17, 1.00874632e-24, ...,
1.65356463e-23, 7.01782132e-21, 1.00000000e+00],
[3.34785675e-08, 7.48466455e-10, 6.04166646e-14, ...,
3.01373211e-13, 1.83766873e-11, 1.00000000e+00],
[2.92747472e-07, 5.75281200e-09, 1.00577006e-14, ...,
2.13655116e-13, 1.68849581e-12, 9.99996543e-01]], dtype=float32)
Utilized Scikit-Learn's classification_report() to get a classification report of the model's performance, having metrics such as Precision, Recall, and F1-score.
Precision: Ratio of true positives to total number of positive predictions [MINIMIZE FALSE POSITIVES]
Recall: Ratio of true positives to total number of actual positive predictions [MINIMIZE FALSE NEGATIVES]
F1 score: Combines precision and recall scores
# Get binary label using np.argmax() to get index. E.g. np.argmax([0,0,0,1]) will return binary value 3
y_test_binary, y_pred_31_binary = np.argmax(y_test, axis=1), np.argmax(y_pred_31, axis=1)
report_31 = classification_report(y_test_binary, y_pred_31_binary, target_names=labels.values())
print(report_31)
precision recall f1-score support
Bean 0.95 0.99 0.97 200
Bitter_Gourd 0.98 0.96 0.97 200
Bottle_Gourd 0.98 0.99 0.99 200
Brinjal 0.92 0.96 0.94 200
Broccoli 0.95 0.96 0.96 200
Cabbage 0.98 0.92 0.95 200
Capsicum 0.97 0.97 0.97 200
Carrot 0.97 0.95 0.96 200
Cauliflower 0.93 0.97 0.95 200
Cucumber 0.96 0.99 0.98 200
Papaya 0.95 0.94 0.95 200
Potato 0.95 0.94 0.94 200
Pumpkin 0.96 0.96 0.96 200
Radish 0.98 0.92 0.95 200
Tomato 0.92 0.93 0.92 200
accuracy 0.96 3000
macro avg 0.96 0.96 0.96 3000
weighted avg 0.96 0.96 0.96 3000
Plotted a confusion matrix to visualize how well the model predicts each label of vegetables.
confusion_matrix(y_test=y_test_binary, y_pred=y_pred_31_binary, title=' of CustomVGG on 31x31 Images')
For the error analysis, I plotted the first 30 incorrect predictions made by the model to visualize where the model went wrong in its vegetable prediction.
error_analysis(y_test=y_test_binary,
y_pred=y_pred_31_binary,
title=' of CustomVGG on 31x31 Images',
X_test=X_test_128,
labels=labels
)
x--------------------------------------------------x Number of incorrect predictions in total: 131 x--------------------------------------------------x
ANALYSIS:
With both our models ready, we can now deploy the model using TensorFlow Serving, which is a serving system for machine learning models designed for production environments, allowing easy deployment of machine learning models.
For TensorFlow Serving, it is required for us to convert our model into a SavedModel format in order to be deployed. Hence, we will be saving the final models here to be used in TensorFlow Serving.
The current timestamp will be utilized in the file path for storing the models in SavedFormat later on.
The timestamp will serve as numerical versioning for the model in the Docker container, allowing for easier differentiation and identification between iterations of models when we choose to update the model further down the line. Furthermore, numerical versioning is compatible with CI/CD pipelines where the tools can programmatically identify the latest version (highest version number in this case by timestamp) to be deployed to production, providing a chronological order of model versions. All in all, it is better practice to implement the timestamp to automatically denote the versions of the models updated compared to manually denoting them.
timestamp = int(time.time())
timestamp
1705840187
file_path_128 = f"./models/conv2d128/{timestamp}"
conv2d_128.save(filepath=file_path_128, save_format='tf')
INFO:tensorflow:Assets written to: ./models/conv2d128/1705840187/assets
INFO:tensorflow:Assets written to: ./models/conv2d128/1705840187/assets
file_path_31 = f"./models/customvgg31/{timestamp}"
customvgg_31.save(filepath=file_path_31, save_format='tf')
INFO:tensorflow:Assets written to: ./models/customvgg31/1705840187/assets
INFO:tensorflow:Assets written to: ./models/customvgg31/1705840187/assets
Now that both of our models have been saved as the SavedModel format, where we can deploy this model using TensorFlow Serving, followed by the deployment on Render.com.
To begin deploying the model, we will have to download the models from the container through the GitLab Repository, and I will store it locally in my C:\ Drive.
GitLab Repository of Model Folder:
Installation will be done by clicking Code on the top right, and downloading the zip folder of the directory, then unzipping it in the C:\ Drive.
Each model's folder should contains these
These are the files required to perform the model's predictions.
Next, we will also have to install the model_config.conf file, as I intend to create a model server for multiple models, and this config file will be utilized to do that.
The installation of the file can be done by clicking the download button on the far middle right.
With those downloaded, we can run the config file and deploy the model in a TensorFlow server from a container using the image tensorflow/serving.
This will be the command we will be running in the cmd prompt:
docker run --name CA2_Models_Serving -p 8501:8501 --mount type=bind,source=C:/CA2_DOAA_MODELS/models/conv2d128,target=/models/conv2d128 --mount type=bind,source=C:/CA2_DOAA_MODELS/models/customvgg31,target=/models/customvgg31 --mount type=bind,source=C:/CA2_DOAA_MODELS/model_config.conf,target=/config/model_config.conf -t tensorflow/serving --model_config_file=/config/model_config.conf
The command differs from typical model deployment using TensorFlow Serving as I am deploying multiple models from one container.
docker run
--name CA2_Model_Serving
-p 8501:8501
--mount type=bind,source=C:/CA2_DOAA_MODELS/models/conv2d128,target=/models/conv2d128
--mount type=bind,source=C:/CA2_DOAA_MODELS/models/customvgg31,target=/models/customvgg31
--mount type=bind,source=C:/CA2_DOAA_MODELS/model_config.conf,target=/config/model_config.conf
-t tensorflow/serving
--model_config_file=/config/model_config.conf
model_config.conf added earlier.After the command has successfully ran, the output should look like this:
And on Docker, there should now be a container called CA2_Models_Serving as shown below
To verify it works, the links to the models are
Conv2D for 128x128 images: http://localhost:8501/v1/models/conv2d128
CustomVGG for 31x31 images: http://localhost:8501/v1/models/customvgg31
Next, we need to set up a network connection between CA2_Models_Server and CA2_Models_Serving as it allows for automated model deployment where when a new model is added to the repository, it can automatically be pushed to the serving container without manual intervention, enabling a smoother CI/CD pipeline for the models further down the road.
To begin, we will set up a network called ca2_dl_network
And connect both containers to the network with docker network connect [network] [container]
Now, we will test the network by pinging it to verify that the connection is up.
To install the ping program if needed, run apt-get update, followed by apt-get install iputils-ping.
To ping the network, run ping CA2_Models_Serving and CRTL+C to stop the pinging.
Next, we will implement unit testing for the models using PyTest to ensure the REST APIs do work in its predictions.
We create the tests folder and the test_docker.py file in order to code the pytests that will be ran to test the APIs.
In order to test the APIs, we will run python -m pytest -v to run the pytests created in the file.
Output:
This shows that our models successfully made the predictions and the APIs do work. Now, we can proceed onto the last step before deploying our models onto Render.
Before we deploy our model, we need a Dockerfile in order to build the image for Render to deploy our models.
Hence, we create the Dockerfile, and code in the commands to assemble the image, as shown below.
Now, it is time to deploy our models!
Head over to dashboard.render.com, and click the "New +" button next to the username on the top right.
Then, select "Web Service" to deploy our models as a web service
Followed by "Build and deploy from a Git repository"
We then connect this repository to render to be utilized for deployment
And fill out the website's information such as name, which in our case is set to CA2_DL_Models, branch etc, and click "Create Web Service"
With that, Render will take care of the rest of the deployment.
After it has successfully been deployed, Render will show that it is "Live" as shown below
We can then head to the two links below to access each of the deep learning models from their respective APIs.
CUSTOMVGG FOR 31x31 IMAGES: https://ca2-dl-models.onrender.com/v1/models/customvgg31
CONV2D FOR 128x128 IMAGES: https://ca2-dl-models.onrender.com/v1/models/conv2d128
Each of the model's website should show:
indicating that it has successfully been deployed and can be utilized in Part B where we develop the web application itself.